Image processing device, electronic camera, image processing method, and image processing program

ABSTRACT

To provide a technique to obtain a synthesized image with an expanded dynamic range with ease and a technique to reduce workload of a user while maintaining the degree of freedom relating to image processing. There provides an image inputting part taking in at least one of a plurality of low-resolution images and a high-resolution image obtained by shooting the same subject while changing exposure condition, a shift detecting part detecting a positional shift of pictorial pattern between each of the plurality of low-resolution images and the high-resolution image, and a gradation expanding part generating a synthesized image in which a range of reproduced gradation is expanded by performing position alignment between each of the plurality of low-resolution images and the high-resolution image based on the positional shift, extracting a gradation information of the plurality of low-resolution images, and synthesizing it with the high-resolution image.

TECHNICAL FIELD

The present application relates to an image processing device, anelectronic camera, an image processing method, and an image processingprogram.

BACKGROUND ART

Conventionally, there is known a technique to generate a synthesizedimage in an expanded dynamic range by shooting the same subject whilechanging exposure settings and synthesizing a plurality of obtainedimages (for example, Patent Document 1).

As a technique to detect a positional shift of pictorial pattern betweenimages output from different imaging devices with different resolutions,the following Patent Document 2 is well known. In this prior art, firstwide-area search is performed between a high-resolution image and alow-resolution image to find a candidate for a matching region. Next, inthe range of the matching region, both images are subjected to pixelinterpolation and a region in which the pictorial pattern matches to thedetail is found by the comparison between interpolated pixels.

Patent Document 1: Japanese Unexamined Patent Application PublicationNo. 2002-305684 Patent Document 2: International Publication WO95/04329Pamphlet DISCLOSURE OF THE INVENTION Problems to be Solved by theInvention

With the prior art in Patent Document 1, however, it is necessary toread a still image with high resolution a plurality of times. Ingeneral, it takes a long time to read an image with high resolution, andtherefore, the times when these images are shot are separatedconsiderably from each other. As a result, if a subject or a cameramoves, the degree of matching of pictorial pattern between a pluralityof images is degraded and it becomes difficult to obtain an excellenteffect for expanding gradation.

On the other hand, on the side of an electronic camera, it is requiredto fix the camera to pin a camera angle from moving during the periodwhich a plurality of still images are shot, and therefore, it isdifficult to perform shooting with expanding gradation with ease.

With the technique disclosed in Patent Document 1, data of a pluralityof images with different exposure conditions described above is recordedin a storage medium in accordance with a choosing operation by a userand then a personal computer reads the image data from the storagemedium to perform synthesizing processing.

As a result, with the technique disclosed in Patent Document 1, it isnecessary for a user to instruct processing to save data of a pluralityof images to be synthesized by performing a choosing operation withconsciousness at the time of shooting in order to perform high precisionimage processing using a personal computer after shooting. Further, whenperforming image processing using a personal computer, it is necessaryto specify image data to be synthesized and subject it to synthesizingprocessing.

Such a choosing instruction is advantageous to users with a high-levelknowledge about image processing in that the degree of freedom aboutimage processing can be improved, however, it is rather a complicatedburden to general users.

The present application is developed with the above-mentioned problemsbeing taken into consideration and a proposition thereof is to provide atechnique to lessen the workload of a user while maintaining thetechnique to easily obtain a synthesized image in an expanded dynamicrange and the degree of freedom about image processing.

Means for Solving the Problems

According to an aspect of the embodiments, an image processing deviceincludes an image inputting part, a shift detecting part, and agradation expanding part as a basic configuration.

The image inputting part takes in at least one of a plurality oflow-resolution images and a high-resolution image obtained by shootingthe same subject while changing exposure condition.

The shift detecting part detects a positional shift of pictorial patternbetween each of the plurality of low-resolution images and thehigh-resolution image.

The gradation expanding part performs position alignment between each ofthe plurality of low-resolution images and the high-resolution imagebased on the positional shift. The gradation expanding part generates asynthesized image in which a range of reproduced gradation is expandedby extracting gradation information of the plurality of low-resolutionimages, and synthesizing it with the high-resolution image.

Preferably, the low-resolution image is the plurality of images. Thegradation expanding part synthesizes the gradation information oflow-resolution represented by each of the plurality of low-resolutionimages with the high-resolution image in a multiplexing manner.

Preferably, the shift detecting part includes a phase dividing part anda precisely detecting part.

The phase dividing part extracts an edge component of pictorial patternfrom the high-resolution image. The phase dividing part generates aplurality of pieces of sampling information with sample positionsshifted from one another by performing sub-sampling of the edgecomponent while shifting phases.

The precisely detecting part detects a positional shift with a finerprecision than a pixel interval of the low-resolution images bydetecting the positional shift with which the pictorial pattern bestmatches between each of the plurality of low-resolution images and theplurality of pieces of sampling information.

Preferably, the gradation expanding part determines a high-brightnessregion and a low-brightness region of the high-resolution image. Thegradation expanding part increases a synthesis ratio of one or morelow-resolution images underexposed as to the high-brightness region ofthe high-resolution image. In addition, the gradation expanding partincreases a synthesis ratio of one or more low-resolution imagesoverexposed as to the low-brightness region of the high-resolutionimage.

More preferably, it may configure the gradation expanding part includedin the above-described first image processing device so as to include anadjusting unit and a controlling unit to be described below.

The adjusting unit adjusts a tone level of a corresponding pixel of asynthesized image by reflecting a tone level of each pixel included inat least one of the low-resolution images chosen from the plurality oflow-resolution images and a tone level of corresponding pixel of thehigh-resolution image. The controlling unit controls adjustingprocessing of the tone level of each pixel of the synthesized image bythe adjusting unit based on at least one of the followings, a histogramof the tone level obtained for each of the plurality of low-resolutionimages, a histogram of the tone level obtained for the high-resolutionimage, and the number of the low-resolution images acquired by the imageinputting unit.

Preferably, in the controlling unit, an analyzing unit analyzes adistribution of pixels in a predetermined range of tone levels as to thehistogram of the tone level obtained for each of the plurality oflow-resolution images and the histogram of the tone level of thehigh-resolution image. A converting curve fitting unit adjusts agradation converting curve used to confine the tone level of each pixelof the synthesized image within a predetermined range in the adjustingunit based on the analysis result of the analyzing unit.

Further preferably, the analyzing unit analyzes a distribution of pixelsin a predetermined range of tone levels as to the histogram of the tonelevel obtained for each of the plurality of low-resolution images andthe histogram of the tone level of the high-resolution image in thecontrolling unit. A choosing unit chooses at least one of thelow-resolution images to be used in adjusting processing by theadjusting unit based on the analysis result by the analyzing unit.

Further preferably, a range determining unit determines the size of aregion of low-resolution images to be reflected in the adjustment of thetone level of each pixel included in the synthesized image by theadjusting unit in accordance with the number of low-resolution images tobe used in synthesizing processing by the adjusting unit in thecontrolling unit.

Further preferably, a brightness weight determining unit determines abrightness weight to be applied when reflecting a brightness component,which corresponds to the pixel of at least one of the low-resolutionimages chosen, to a brightness component of each pixel included in thesynthesized image by the adjusting unit in the controlling unit. Acolor-difference weight determining unit determines a color-differenceweight to be applied when reflecting a color-difference component, whichcorresponds to the pixel of at least one of the low-resolution imageschosen, to a color-difference component of each pixel included in thesynthesized image by the adjusting unit. In the color-difference weightdetermining unit, the weight adjusting unit adjusts a value of thecolor-difference weight in accordance with the magnitude of a brightnesscomponent corresponding to each pixel of the high-resolution image.

An electronic camera disclosed below includes the basic components ofthe image processing device described above and an imaging part thatshoots a subject with at least two kinds of resolution. In this case,the high-resolution image processed by the image processing device is astill image of the high-resolution shot by the imaging part. The imageprocessing device processes at least one of the plurality of thelow-resolution images shot by the imaging part before and/or after theshooting of the still image with the exposure condition different fromthe high-resolution image.

Preferably, there provides a monitor part that displays an image. Theimaging part sequentially shoots low-resolution through images, throughimages are images obtained by pixel skipping to provide moving imagesfor view finder, and displays a moving image on the monitor part.Further, the imaging part shoots the plurality of low-resolution imageswith timing not synchronizing the shooting of the through images andunder the exposure condition different from the high-resolution image.

According to another aspect of the embodiment, the image processingdevice is configured to include a determining unit determines whether ornot to attach one or more pieces of other image data as auxiliaryinformation when performing image processing to an image data based onthe image data capturing an image of a subject shot by an imaging unitand a predetermined condition, and a recording unit puts together a mainimage data, which is the image data to be processed in the imageprocessing, and the auxiliary information into one image file andrecords in a recording medium in accordance with the determinationresult of attaching the auxiliary information.

Preferably, the configuration is such that the above-describeddetermining unit includes a saturation detecting unit notifying therecording unit of the determination result to attach the auxiliaryinformation when detecting a region in which a tone level of pixelincluded in the main image data is saturated.

Preferably, the configuration is such that the recording unit includes afirst choosing unit that chooses other image data, obtained by shootingthe same subject as the main image data at minimal time intervals, asthe auxiliary information and used in process of attaching to the mainimage data.

Preferably, the configuration is such that the recording unit includes asecond choosing unit that chooses other image data, obtained by shootingthe same subject as the main image data at minimal time intervals undera different shooting condition, as the auxiliary information and used inprocess of attaching to the main image data.

Preferably, the configuration is such that the recording unit includes athird choosing unit that chooses image data in which a distributionrange of a tone level of pixel represented in a histogram has apredetermined relationship with a peak position in a histogram of themain image data among other images, obtained by shooting the samesubject as the main image data at minimal time intervals under adifferent shooting condition, as the auxiliary information and used inprocess of attaching to the main image data.

Preferably, the configuration is such that the recording unit includes aheader creating unit that creates header information including theauxiliary information and attaches it to the main image data.

Further preferably, the configuration is such that the above-describedheader creating unit includes an extracting unit that extracts part ofother image data used in image processing of the main image data inaccordance with the purpose of the image processing and providing thepart of other image data extracted in header information creatingprocessing as auxiliary information.

According to an aspect of the embodiments, an image processing methodconfigures as follows.

In an image inputting step, a plurality of low-resolution imagesobtained by shooting the same subject under a plurality of shootingconditions with an imaging unit that shoots images of the subject with aplurality of different resolutions and a high-resolution image obtainedby shooting the subject under a correct exposure condition with theimaging unit are acquired. In a synthesizing step, at least one of thelow-resolution images chosen from the plurality of low-resolution imagesand the high-resolution image are synthesized to generate a synthesizedimage having a resolution equivalent to the high-resolution image. Inthis synthesizing step, a tone level of a corresponding pixel of thesynthesized image is adjusted by reflecting a tone level of each pixelincluded in at least one of the low-resolution images chosen and a tonelevel of corresponding pixel of the high-resolution image in anadjusting step. In a controlling step, the adjusting processing of thetone level of each pixel of the synthesized image in the adjusting stepis controlled based on at least one of the followings, a histogram of atone level obtained for each of the plurality of low-resolution images,a histogram of a tone level obtained for the high-resolution image, andthe number of the low-resolution images acquired in the image inputtingstep.

According to another aspect of the embodiments, the image processingmethod below, main image data and auxiliary information used in imageprocessing are read from an image file including the main image data,which to be processed in an image processing, recorded in acomputer-readable recording medium and the image processing for the mainimage data is performed using the read auxiliary information.

The above-described image processing device can also be realized bycausing a computer to function as the above-described image processingdevice with an image processing program.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an electronic camera 10(including an image processing device 25) in the present embodiment.

FIG. 2 is a block diagram schematically illustrating a configuration ofthe image processing device 25.

FIG. 3 is a flow chart (1/2) that explains the operation of theelectronic camera 10.

FIG. 4 is a flow chart (2/2) that explains the operation of theelectronic camera 10.

FIG. 5 is a flow chart that explains a shooting sequence of theelectronic camera 10.

FIG. 6 are a diagram that explains a detecting procedure of a positionalshift.

FIG. 7 is a diagram that explains sub-sampling of a high-resolutionimage.

FIG. 8 is a diagram that explains the generation of a rearranged image.

FIG. 9 is a diagram that explains gradation conversion of a synthesizedimage.

FIG. 10 is a diagram that explains an effect for expanding gradation.

FIG. 11 is a diagram illustrating another embodiment of an imageprocessing device according to the present application.

FIG. 12 is a flow chart representing image synthesizing processing.

FIG. 13 are a diagram that explains distribution analyzing processingbased on a histogram.

FIG. 14 are an explanatory diagram of a gradation converting curve.

FIG. 15 is a diagram illustrating another embodiment of an imageprocessing device according to the present application.

FIG. 16 is a diagram illustrating an embodiment of an image processingdevice according to the present application.

FIG. 17 is a diagram illustrating a configuration of an image file.

FIG. 18 are a diagram illustrating a relationship of correspondencebetween a high-resolution image and a through image.

FIG. 19 is a diagram illustrating an embodiment of an image processingmethod according to the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present application will be explained indetail based on the drawings.

Embodiment 1 Explanation of Electronic Camera Configuration

FIG. 1 is a block diagram illustrating an electronic camera 10(including an image processing device 25) in the present embodiment.

In FIG. 1, a shooting lens 12 is mounted on the electronic camera 10. Inan image space of the shooting lens 12, an imaging surface of an imagingdevice 11 is arranged. The imaging device 11 includes a mode in which ahigh-resolution image is read and a mode in which a low-resolution imageis read by performing pixel skipping and pixel data accumulation withinthe device. These reading modes are controlled by an imaging controllingpart 14. An image signal generated by the imaging device 11 is storedtemporarily in a memory 17 after being processed via a signal processingpart 15 and an A/D converting part 16.

The memory 17 is coupled to a bus 18. To the bus 18, the imagingcontrolling part 14, a microprocessor 19, a recording part 22, an imagecompressing part 24, a monitor display part 30, the image processingdevice 25, etc., are also coupled. To the above-mentioned microprocessor19, an operation part 19 a, such as a release button, is coupled.Further, to the above-mentioned recording part 22, a storage medium 22 ais mounted detachably.

[Explanation of Image Processing Device 25]

FIG. 2 is a block diagram schematically illustrating a configuration ofthe image processing device 25.

A gain correcting part 31 performs gain correction and gradationcorrection for an image in the memory 17. A high-resolution image readfrom the memory 17 is transferred to a reduced image making part 32, afeature extraction part 33, and a gradation synthesizing part 34 b,respectively. The output data of the reduced image making part 32 istransferred to a coarse detecting part 36 via a feature extraction part35. The output data of the feature extraction part 33 is transferred toa precisely detecting part 38 via a phase dividing part 37. Informationabout an edge is transferred from the feature extraction part 33 to thegain correcting part 31.

On the other hand, a plurality of low-resolution images read from thememory 17 is transferred to a feature extraction part 39 and a positionaligning part 34 a, respectively. The output data of the featureextraction part 39 is transferred to the coarse detecting part 36 andthe precisely detecting part 38, respectively.

A positional shift coarsely detected by the coarse detecting part 36 istransferred to the precisely detecting part 38. A positional shiftprecisely detected by the precisely detecting part 38 is transferred tothe position aligning part 34 a. The position aligning part 34 a adjuststhe position of pixel of a low-resolution image based on the positionalshift and outputs it to the gradation synthesizing part 34 b. Thegradation synthesizing part 34 b acquires information about the amountof gain correction from the gain correcting part 31 and synthesizes aplurality of low-resolution images with a high-resolution image based onthe acquired information.

The gradation expanding part according to claims corresponds to theposition aligning part 34 a and the gradation synthesizing part 34 b.

[Explanation of Operation]

FIG. 3 and FIG. 4 are flow charts that explain the operation of theelectronic camera 10. Hereinafter, the operation is explained along stepnumbers represented in these drawings.

Step S1: When the main power source of the electronic camera 10 isturned on, the microprocessor 19 instructs the imaging controlling part14 to read a through image. The imaging controlling part 14 drives theimaging device 11 in a low-resolution reading mode and sequentiallyreads through images at, for example, 30 frames per second, asrepresented in FIG. 5.

Step S2: The through image read from the imaging device 11 is processedvia the signal processing part 15 and the A/D converting part 16. Amongthe through images, those in which the tone shift by an exposureadjustment, to be described later, is small are displayed in a movingimage on the monitor display part 30. It is possible for a user todetermine a picture composition of shooting with the electronic camera10 by viewing the display of the moving image of through images.

On the other hand, the microprocessor 19 performs exposure calculationbased on the result of photometry by a photometer part (not represented)and the brightness of the through image and determines an exposurecondition of a high-resolution image.

Step S3: The imaging controlling part 14 drives the imaging device 11 ina low-resolution reading mode during the period of shooting of throughimages. As a result, one or more through images (low-resolution images)are generated as represented in FIG. 5.

At this time, the imaging controlling part 14 adjusts the exposurecondition of the low-resolution image so that the range of reproducedgradation is different from that of the high-resolution image. Forexample, compared to the exposure time of the high-resolution image, theexposure time of the low-resolution image is varied to two-levelunderexposed, one level underexposed, one level overexposed, andtwo-level overexposed. The low-resolution images thus generated arestored temporarily in the memory 17.

When the number of recorded low-resolution images exceeds apredetermined upper limit number, the imaging controlling part 14deletes the low-resolution images from older one. It is preferable todetermine in advance the upper limit number in accordance with theavailable storage capacity of the memory 17 etc.

Step S4: Here, the microprocessor 19 determines whether or not thefull-pressing operation of a release button is performed by a user.

When the full-pressing operation of the release button is performed, themicroprocessor 19 moves the operation to step S5.

On the other hand, when the full-pressing operation of the releasebutton is not performed, the microprocessor 19 returns the operation tostep S1.

Step S5: Here, the microprocessor 19 determines whether or not theexposure time of the high-resolution image determined in step S2 isequal to or below an allowable upper limit which camera shake is notremarkable in image. For example, the allowable upper limit is set toabout 1/(focal length of the shooting lens 12 converted into 35 mm size)seconds.

When the exposure time setting is less than or equal to the allowableupper limit, the microprocessor 19 moves the operation to step S6.

On the other hand, when the exposure time setting exceeds the allowableupper limit, the microprocessor 19 moves the operation to step S7.

Step S6: The imaging controlling part 14 performs shutter control forthe imaging device 11 in accordance with the set exposure time.Subsequently, the imaging controlling part 14 drives the imaging device11 in the high-resolution reading mode and reads the high-resolutionimage. The high-resolution image (still image) is stored temporarily inthe memory 17 via the signal processing part 15 and the A/D convertingpart 16.

After this operation, the microprocessor 19 moves the operation to stepS9.

Step S7: On the other hand, when it is determined that the exposure timesetting exceeds the allowable upper limit against camera shake, themicroprocessor 19 limits the exposure time equal to or below theallowable upper limit which camera shake is not remarkable in image.

The imaging controlling part 14 performs shutter control for the imagingdevice 11 in accordance with the exposure time limited to a shorter one.In this state, the imaging controlling part 14 drives the imaging device11 in the high-resolution reading mode and reads the high-resolutionimage. The high-resolution image is an image the signal level of whichis low because of underexposure, however, in which camera shake is lessconspicuous. The high-resolution image is stored temporarily in thememory 17.

Step S8: The gain correcting part 31 performs gain correction of theunderexposed high-resolution image.

Step S9: The gain correcting part 31 determines whether or not thehigh-resolution image and the low-resolution image in the memory 17 havealready been subjected to gamma correction. For the image having alreadybeen subjected to gamma correction, the gain correcting part 31 performsinverse gamma correction (at this time, it is preferable to increasequantifying bit number for gradation component so that the gradationwidth of the image is not limited substantially).

This processing makes it possible to perform synthesizing processing ofan image, to be described later, on a substantially linear gradationaxis.

Step S10: The feature extraction part 33 takes in the high-resolutionimage and extracts a vertical edge component gv and a horizontal edgecomponent gh using an edge extraction filter.

Here, it is preferable to switch the edge extraction filters as followsin accordance with the reading method of the low-resolution image.

When the low-resolution image is created by pixel data accumulation orpixel averaging

gv(x,y)=[−f(x,y−4)−f(x,y−3)−f(x,y−2)−f(x,y−1)+f(x,y+4)+f(x,y+5)+f(x,y+6)+f(x,y+7)]/4

gh(x,y)=[−f(x−4,y)−f(x−3,y)−f(x−2,y)−f(x−1,y)+f(x+4,y)+f(x+5,y)+f(x+6,y)+f(x+7,y)]/4

When the low-resolution image is created by pixel skipping

gv(x,y)=−f(x,y−4)+f(x,y+4)

gh(x,y)=−f(x−4,y)+f(x+4,y)

In order to reduce the influence of noise, it is preferable for thefeature extraction part 33 to replace the vertical edge component gv andthe horizontal edge component gh that fall within a predeterminedreduced amplitude with zero.

The feature extraction part 33 chooses a region in which the number ofedge components is large in the image based on the edge components gv,gh and determines the region as a target region.

Step S11: The brightness level of the low-resolution image is differentfrom the brightness level of the high-resolution image because of thedifference in the exposure condition. Therefore, the gain correctingpart 31 performs gain correction for the low-resolution image in thememory 17 to adapt the brightness level to that of the high-resolutionimage.

For example, for the low-resolution image that has been subjected ton-stage exposure correction, its brightness level can be adapted to thatof the high-resolution image by multiplying the linear brightness levelby a factor of 1/(2^(n)).

It may also be possible to perform the gain correction of thelow-resolution image so that the brightness level of the low-resolutionimage matches with that of the high-resolution image in the targetregion obtained in step S10.

Step S12: The reduced image making part 32 adapts the number of pixelsof the high-resolution image to that of the low-resolution image byconverting the resolution of the high-resolution image after gainadjustment.

For example, it is possible to convert resolution so that the numbers ofvertical and horizontal pixels of the high-resolution image are reducedto ¼, respectively, by extracting an average value of 4×4 pixels for thecorresponding pixel of the reduced image.

The image the resolution of which is reduced as described above(hereinafter, referred to as a reduced image) is transferred to thefeature extraction part 35.

Step S13: Subsequently, the coarse detecting part 36 detects apositional shift between the reduced image and the low-resolution image.

FIG. 6 are a diagram illustrating a procedure for finding a positionalshift by comparing edge projections. Hereinafter, processing to detect apositional shift at a high speed is explained using FIG. 6.

First, the feature extraction part 35 extracts a vertical edge componentgv′ from a reduced image f (x, y) using a vertical edge extractionfilter represented in the following expression (refer to FIG. 6[A]).

gv′(x,y)=−f(x,y−1)+f(x,y+1)

Further, the feature extraction part 35 extracts a horizontal edgecomponent gh′ from the reduced image f(x, y) using a horizontal edgeextraction filter represented in the following expression (refer to FIG.6[B]).

gh′(x,y)=−f(x−1,y)+f(x+1,y)

In order to reduce the influence of noise, it is preferable for thefeature extraction part 35 to replace the vertical edge component gv′and the horizontal edge component gh′ that fall within a small enoughpredetermined amplitude with zero.

Next, the feature extraction part 35 calculates a vertical projectionprofile by accumulating the vertical edge component gv′ in units ofhorizontal rows as represented in FIG. 6[A].

Further, the feature extraction part 35 calculates a horizontalprojection profile by accumulating the horizontal edge component gh′ inunits of vertical columns as represented in FIG. 6[B].

On the other hand, the feature extraction part 39 takes in a pluralityof low-resolution images from the memory 17. The feature extraction part39 performs the same processing as the feature extraction part 35 to theindividual low-resolution images to find a vertical projection profileand a horizontal projection profile, respectively.

Here, the coarse detecting part 36 finds a difference by shifting thevertical projection profile in the center region of the reduced imageand the vertical projection profile in the center region of thelow-resolution image as represented in FIG. 6[A] and detects a profileshift with which the sum of the absolute differences is the minimum.This profile shift corresponds to the positional shift between thereduced image and the low-resolution image in the vertical direction.

Further, the coarse detecting part 36 finds a difference by shifting thehorizontal projection profile in the center region of the reduced imageand the horizontal projection profile in the center region of thelow-resolution image as represented in FIG. 6[B] and detects a profileshift with which the sum of the absolute differences is the minimum.This profile shift corresponds to the positional shift between thereduced image and the low-resolution image in the horizontal direction.

In this manner, the coarse detecting part 36 finds the positional shifts(coarse detection result) of the plurality of low-resolution images withthe reduced image as a positional reference, respectively, and outputsthem to the precisely detecting part 38.

Step S14: Next, the positional shift between the high-resolution imageand the low-resolution image is detected precisely.

First, the feature extraction part 33 calculates a vertical projectionprofile of the high-resolution image by accumulating the vertical edgecomponent gv obtained in step S10 in units of horizontal rows. Further,the feature extraction part 33 also calculates a horizontal projectionprofile of the high-resolution image by accumulating the horizontal edgecomponent gh obtained in step S10 in units of vertical columns.

The phase dividing part 37 performs sub-sampling of the verticalprojection profile of the high-resolution image for every four pixels.At this time, the phase dividing part 37 generates four kinds ofsampling information the phases of which are shifted from each other asrepresented in FIG. 7 by shifting the phase of sub-sampling.

Similarly, the phase dividing part 37 performs sub-sampling of thehorizontal projection profile of the high-resolution image for everyfour pixels. At this time, the phase dividing part 37 generates fourkinds of sampling information the phases of which are shifted from eachother by shifting the phase of sub-sampling.

Step S15: The precisely detecting part 38, starting from the positionalshift which is the result of the coarse detection process performed bythe coarse detecting part 36, detects a profile shift with which the sumof absolute differences is the minimum by finding a difference whileshifting the sampling information of the vertical projection profileobtained from the high-resolution image and the vertical projectionprofile of the low-resolution image.

The precisely detecting part 38 finds a profile shift with which thecharacteristics of pictorial pattern (here, profile) best match byperforming the detection of the profile shift for the four kinds ofsampling information, respectively. This profile shift corresponds tothe positional shift in the horizontal direction. Further, the preciselydetecting part 38 detects a positional shift in the vertical directionin a similar manner.

As described above, the precisely detecting part 38 finds the positionalshifts (precise detection result) of the plurality of low-resolutionimages with the high-resolution image as the positional reference with aprecision finer than the pixel interval of the low-resolution image andoutputs them to the position aligning part 34 a.

Step S16: The position aligning part 34 a expands the low-resolutionimage (magnification of 4×4). At this time, the position aligning part34 a obtains an expanded image with expanded pixel interval withoutperforming pixel interpolation.

Next, the position aligning part 34 a performs mapping (rearrangement)as represented in FIG. 8 by respectively displacing the pixel positionsof the expanded image of the low-resolution image based on the precisedetection result of the positional shift obtained by the preciselydetecting part 38. In this manner, it is possible to obtain a rearrangedimage having substantially the same numbers of vertical and horizontalpixels as those of the high-resolution image.

Step S17: In the rearranged image for which the mapping processing iscompleted, there remains pixels unmapped, pixels shifted from the normalpixel position, and overlapped pixels.

Therefore, the position aligning part 34 a picks up a nearby pixel foreach normal pixel position of the rearranged image. The positionaligning part 34 a applies Gaussian filter in the following expressionto the color-difference component of these nearby pixels.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 1} \right\rbrack & \; \\{{{g\left( {i,j} \right)} = \frac{\left\lbrack {\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}\left\{ {{G\left( {k,l} \right)}{f\begin{pmatrix}{{i - {\left( {m - 1} \right)/2} + k - 1},} \\{j - {\left( {m - 1} \right)/2} + l - 1}\end{pmatrix}}} \right\}}} \right\rbrack}{\left\{ {\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}{G\left( {k,l} \right)}}} \right\}}}{{G\left( {i,j} \right)} = {\exp\left\lbrack {- \frac{i^{2} + j^{2}}{2\sigma^{2}}} \right\rbrack}}} & \;\end{matrix}$

The f (x, y) is the color-difference component of a pixel position (x,y) of the rearranged image. Here, m is a range of the nearby pixels andσ is a numerical value to adjust a weighted ratio. For example, asspecific numerical values, it is preferable that m=5 and σ=3.

The position aligning part 34 a determines the color-differencecomponent of the rearranged image by regarding the calculation result ofthe Gaussian filter as the color-difference component of the normalpixel position.

Step S18: The gradation synthesizing part 34 b performs the next filterprocessing for the brightness component of the high-resolution image.First, the gradation synthesizing part 34 b extracts the brightnesscomponent from the high-resolution image after having been subjected togain correction and performs filter processing that combines medianprocessing and the Gaussian filter. For example, the gradationsynthesizing part 34 b sets the filter size to 3×3 pixels, extractsthree medians from the nine pixels in this filter size, and thenperforms the Gaussian filter. Due to this processing, it is possible toreduce the amount of noise produced in the brightness componentresulting from underexposure etc.

Step S19: The gradation synthesizing part 34 b extracts an image regionhaving a high possibility of being saturated gradation as mapinformation from the high-resolution image. For example, the gradationsynthesizing part 34 b extracts an image region the brightness level ofwhich reaches the saturation level of the imaging device 11 as ahigh-brightness region. In addition, for example, the gradationsynthesizing part 34 b extracts an image region the brightness level ofwhich does not exceed the range of saturated black of the imaging device11 as a low-brightness region.

Step S20: The gradation synthesizing part 34 b obtains information aboutthe amount of gain correction of the low-resolution image performed instep S11 from the gain correcting part 31. According to the amount ofgain correction, the gradation synthesizing part 34 b determines thesynthesis ratio between the high-resolution image and the low-resolutionimage as follows.

(1) Low-brightness region of high-resolution image . . . The synthesisratio of the low-resolution image applying the amount of gain correctionwhich is less than 1 is set as high as about, for example, 30 to 50%. Onthe other hand, as to the low-resolution image the amount of gaincorrection of which is more than 1, the synthesis ratio is set as low asabout 0%. Due to this setting, it is possible to reflect the gradationinformation represented by dark part of the low-resolution image havingbeen subjected to overexposure correction in the high-resolution image.

(2) High-brightness region of high-resolution image . . . The synthesisratio of the low-resolution image the amount of gain correction of whichis more than 1 is set as high as about, for example, 30 to 50%. On theother hand, as to the low-resolution image the amount of gain correctionof which is less than 1, the synthesis ratio is set as low as about 0%.Due to this setting, it is possible to reflect the gradation informationrepresented by bright part of the low-resolution image having beensubjected to underexposure correction in the high-resolution image.

(3) Edge of high-resolution image . . . A portion (edge) is extracted,where the brightness component locally changes so as to exceed apredetermined ratio. The synthesis ratio of the low-resolution image forthis edge is reduced as low as about 0%. Due to this setting, it ispossible to maintain the edge structure of the high-resolution image.

Step S21: The position aligning part 34 a corrects the positional shiftof the low-resolution image in accordance with the precise detectionresult of the positional shift detected in step S15. The gradationsynthesizing part 34 b extracts the brightness component from thelow-resolution image after having been subjected to positional shiftcorrection and performs weighted addition to the brightness component ofthe high-resolution image in accordance with the synthesis ratiodetermined in step S20. Due to this weighted addition, the range ofreproduced gradation of the synthesized image is expanded substantiallyin the linear manner.

Step S22: The gradation synthesizing part 34 b performs gradationcorrection of the brightness component of the synthesized image usingthe gradation converting characteristics as represented in FIG. 9. Withthe gradation converting characteristics, the gradation range of medium-to high-brightness is subjected to gradation compression, and thereby,the gradation information restored the high-brightness region isconfined within a range of practical signal values. Further, as to therange of low-brightness gradations, gradation extension is performed toan extent at which saturated black is suppressed and dark part noisedoes not increase, and thereby, the gradation information restored inthe low-brightness region is confined within a range of practical signalvalues.

Step S23: By combining the color-difference component (rearranged image)generated in step S17 and the brightness component (synthesized image)generated in step S22, a color image in which the range of reproducedgradation is expanded is completed. This color image is recorded andstored in the recording medium 22 a via the image compressing part 24,the recording part 22, etc.

Effects and Others of the Embodiment

In the present embodiment, a low-resolution image generated in thelow-resolution reading mode is utilized for expanding the gradation of astill image (high-resolution image). The low-resolution image is read ata high rate of, for example, 30 to 60 frames per second. As a result,when synthesizing the high-resolution image and the low-resolutionimage, collapse resulting from the difference in pictorial pattern isunlikely to occur, and therefore, it is possible to obtain an excellenteffect for expanding gradation.

Further, in the present embodiment, a plurality of low-resolution imagesis generated while correcting exposure in the positive and negativedirections. Consequently, it is made possible to obtain gradationinformation in various gradation regions. As a result, it is possible toobtain an excellent effect for expanding gradation in thehigh-brightness and low-brightness gradation regions.

Further, even if a plurality of low-resolution images is shot asdescribed above, it is possible to continuously shoot at a high framerate, and therefore, shooting is completed in an instant. Consequently,the period of time during which a user has to fix a camera angle isshortened considerably than before. As a result, it is possible for auser to perform shooting with expanding gradation with ease.

Further, in the present embodiment, a plurality of pieces of samplinginformation in accordance with the sampling phases which are shiftedfrom one another is generated from the high-resolution image. Bydetecting a positional shift between the sampling information and thelow-resolution image, it is possible to detect a positional shift with aprecision finer than the pixel interval of the low-resolution image. Asa result, it is made possible to further improve the position aligningprecision of pictorial pattern between the high-resolution image and thelow-resolution image, and therefore, it is possible to obtain a moreexcellent effect for expanding gradation.

Further, in the present embodiment, as to the region where spots ofsaturated gradations are few, the synthesis ratio of the low-resolutionimage is reduced adaptively. Consequently, it is also made possible tokeep the original gradation information of the high-resolution imagefaithfully.

Further, in the present embodiment, at the edge of the high-resolutionimage, the synthesis ratio of the low-resolution image is reducedlocally. Consequently, it is possible to avoid troubles, such as thatedge is turned into multiple lines after synthesis.

Complementary Items of Embodiment

The inventors of the present application have disclosed the procedure tofurther increase the speed of the positional shift detection in JapanesePatent Application No. 2005-34571. It may also be possible to increasethe speed of the positional shift detection in the present embodimentaccording to the procedure.

In step S13, the absolute positional shift between the reduced image ofthe high-resolution image and the low-resolution image is detectedcoarsely. However, the present application is not limited to this. Itmay also be possible to coarsely detect the relative positional shiftbetween a plurality of low-resolution images. It is possible for theprecisely detecting part 38 to roughly estimate the remaining absolutepositional shift based on the relatively coarse detection result and theprecise detection result of at least one positional shift. It is madepossible for the precisely detecting part 38 to quickly detect theprecise positional shift by searching for a positional shift with theabsolute coarse detection result as its start point.

In the above-described embodiment, the positional shift of an image isdetected from the comparison between projection profiles. However, thepresent application is not limited to this. For example, it may also bepossible to detect the positional shift by the spatial comparisonbetween pixel arrangements of both images.

In the above-described embodiment, the case is explained where the imageprocessing device 25 is mounted on the electronic camera 10. However,the present application is not limited to this. It may also be possibleto create an image processing program into which the above-describedimage processing is encoded. It is made possible to effectively utilizegradation information of a low-resolution image to expand the gradationof a high-resolution image by causing a computer to execute the imageprocessing program.

In the above-described embodiment, the low-resolution image is obtainedbefore the shooting of the high-resolution image. However, the presentapplication is not limited to this.

For example, it may also be possible to obtain the low-resolution imageafter the shooting of the high-resolution image. Further, it may also bepossible to obtain a plurality of low-resolution images over the periodof time before and after the shooting of the high-resolution image.

In the above-described embodiment, the gradation information of thelow-resolution image is provided for both the high-brightness region andthe low-brightness region. However, the present application is notlimited to this. The gradation information of the low-resolution imagemay be provided for one of the high-brightness region and thelow-brightness region. For example, it is possible to expand the rangeof reproduced gradation in the high-brightness region by providing thegradation information of the low-resolution image having been subjectedto the negatively-exposed correction for the high-brightness region ofthe high-resolution image. In addition, it is possible to expand therange of reproduced gradation in the low-brightness region by providingthe gradation information of the low-resolution image having beensubjected to the positively-exposed correction for the low-brightnessregion of the high-resolution image.

In the above-described embodiment, the case is explained where the imagesignal of the brightness color-difference is dealt with. However, thepresent application is not limited to this. In general, it may also bepossible to apply the present application to a case where RGB, Lab, orother image signals are dealt with.

In the above-described embodiment, the positional shift of pictorialpattern is detected by image processing. However, the presentapplication is not limited to this. For example, it may also be possibleto obtain movement (oscillation) of the shooting region of the camera bymounting an acceleration sensor etc. on the camera side and detect thepositional shift of pictorial pattern of a plurality of images from themovement (oscillation) of the shooting region.

In the above-described embodiment, only the low-resolution images theexposure conditions of which have been changed are synthesized with thehigh-resolution image. However, the present application is not limitedto this. For example, it may also be possible to include alow-resolution image the exposure condition of which is the same as thatof the high-resolution image in synthesis. The low-resolution image withthe same exposure condition has an effect to, for example, improve S/Nof a synthesized image.

As described above, the image processing device disclosed in the sectionof MEANS FOR SOLVING PROBLEM, a low-resolution image and ahigh-resolution image having different exposure conditions aresynthesized and thereby the range of reproduced gradation is expanded.In this case, as to the low-resolution image, the number of pixels issmall, and therefore, the read time during the period of shooting can beshortened. Consequently, it is possible to increase the degree ofmatching of pictorial pattern between images by shortening the intervalbetween the shooting time of the high-resolution image and thelow-resolution image. As a result, the pictorial patterns match wellwith each other when synthesizing an image, and therefore, it ispossible to obtain an excellent effect for expanding gradation.

With the electronic camera disclosed also in the section of means forsolving problem, a high-resolution image and a low-resolution image areshot under different exposure conditions. In this case, the read time ofthe low-resolution image is short, and therefore, it is possible tocomplete the shooting of the low-resolution image in a brief time. As aresult, the period during which a user has to fix the camera angle isshortened and easier shooting with expanding gradation is enabled.

Embodiment 2

FIG. 11 represents a second embodiment of an image processing deviceaccording to the present application.

Among components represented in FIG. 11, those equivalent to thecomponents represented in FIG. 1 or FIG. 2 are representing the samesymbols assigned and their explanation is omitted.

In a digital camera represented in FIG. 11, light formed into a opticalimage on the imaging device 11 by the shooting optical system 12 whenshooting an image is converted into an electric signal according to itsintensity by the imaging device 11, and is further converted intodigital data by an analog/digital (A/D) converter 23 and stored in thememory 17. The memory 17 represented in FIG. 11 is coupled with theimage processing device 25, the image compressing part 24, the recordingpart 22, and a shooting controlling part 28 via a bus and the shootingcontrolling part 28 switches the reading mode of the imaging device 11.

The shooting controlling part 28 represented in FIG. 11 instructs thehigh resolution mode to the above-described imaging device 11 to readdata corresponding to all of the pixels in response to the operation ofthe release button by a user and the high-resolution image data obtainedfrom the electric signal read by the imaging device 11 in response tothis is stored in the memory 17 and, at the same time, subjected to theprocessing of the image processing device 25. On the other hand, beforeand after the release button is operated, the shooting controlling part28 switches the reading mode of the imaging device 11 to the throughimage mode and in response to this, the low-resolution image dataobtained by pixel skipping and pixel data accumulation within theimaging device 11 is subjected to display processing by a display part29 via the memory 17 and thus the user is provided with informationabout the shooting range.

During the period of time in which the above-described through imagemode is applied, the low-resolution image data obtained from the outputsignal of the imaging device 11 obtained under various exposureconditions is stored in the memory 17 and subjected to the processing ofthe image processing device 25 along with the above-describedhigh-resolution image data. The image data having been subjected to theprocessing of the image processing device 25 is transferred to therecording part 22 via a bus after being compressed by the imagecompressing part 24 and recorded in the recording medium 22 a.

In the image processing device 25 represented in FIG. 11, a positionalignment processing part 42 extracts features from the high-resolutionimage and the plurality of low-resolution images received from thememory 17 and the positional shift between them is corrected based onthe extracted features. Further, the gain correcting part 31 representedin FIG. 11 performs gain correction in accordance with the difference inthe exposure condition between the high-resolution image and theindividual low-resolution images and the low-resolution image after thecorrection is synthesized with the high-resolution image based on theprocessing result of the above-described position alignment processingpart 42 by the gradation synthesizing part 34 b, and subjected to theprocessing of the image compressing part 24.

The above-described position alignment processing part 42 is configuredby each part that provides functions relating to the correctionprocessing of positional shift in the image processing device 25represented in FIG. 2.

In the image processing device 25 represented in FIG. 11, a distributionanalyzing part 44 creates histograms of tone level, respectively, forthe high-resolution image and the low-resolution image corrected by thegain correcting part 31, and analyzes the histograms and subjects theanalysis result to the processing of a synthesis controlling part 45.

The synthesis controlling part 45 represented in FIG. 11 determinesvarious parameters to be applied to the gradation synthesizingprocessing in the gradation synthesizing part 34 b based on the analysisresult of the distribution analyzing part 44 and subjects theseparameters to the processing of the gradation synthesizing part 34 b,and thus controlling the processing of the gradation synthesizing part34 b.

Hereinafter, the detailed operations of the distribution analyzing part44 and the synthesis controlling part 45 are explained by taking a caseas an example, where the high-resolution image with correct exposureobtained by the present shooting and the through images (low-resolutionimages) obtained with one level underexposed and two levels underexposedare synthesized to generate a high-resolution image with an expandeddynamic range in which the range of reproduced gradation is expanded.

FIG. 12 is a flow chart representing image synthesizing processing. FIG.13 represent a diagram that explains distribution analyzing processing.

From among the through images stored in the memory 17, a through imageapplied with an exposure value lower than that of the exposure conditionapplied to the present shooting is extracted and read by the imageprocessing device 25 along with the high-resolution image obtained bythe present shooting (step S31). In the memory 17, through imagesobtained under various exposure conditions are stored for the exposurevalue determining processing prior to the present shooting and fromamong these through images, for example, a through image obtained withone level underexposed and a through image obtained with two levelsunderexposed with respect to the correct exposure applied to the presentshooting are read from the memory 17 and subjected to subsequentprocessing.

For the through images read in this manner, the tone value of each pixelincluded in the through image is multiplied by a constant in accordancewith the ratio between the exposure value applied to each of the throughimages and the correct exposure by the gain correcting part 31, and thusgain correction is performed (step S32). Here, if the read through imagehas already been subjected to gamma transformation, inverse gammatransformation is performed prior to gain correction. Consequently, thehistogram of the through image obtained with one level underexposed(refer to FIG. 13( b)) and the histogram of the through image obtainedwith two levels underexposed (refer to FIG. 13( d)) are converted intohistograms (refer to FIGS. 13( c), (e)) that can be compared with thehistogram of the high-resolution image shot with correct exposure (referto FIG. 13( a)) in terms of the distribution of pixels in a linearspace.

For the histogram of the through image thus obtained, the distributionanalyzing part 44 searches for a range where more than a predeterminedthreshold number of pixels are distributed in a high-brightness range,which is equivalent to saturated level when correct exposure is applied(The high-brightness range is more than or equal to the tone levelindicated by the broken line in FIG. 13). The distribution analyzingpart 44 extracts the range obtained by searching as a feature of thegradation in the high-brightness region (step S3).

In step S3, for example, when ranges indicated surrounded by the brokenline in FIGS. 13( c), (e) are detected, the distribution analyzing part44 notifies the synthesis controlling part 45 that the feature of thegradation in the high-brightness region is extracted and in response tothis, the synthesis controlling part 45 chooses the through image fromwhich the feature of the gradation has been extracted as alow-resolution image to be synthesized (step S34).

When performing synthesizing processing, the distribution analyzing part44 finds the total number of pixels in the high-brightness region havinga tone level more than or equal to a predetermined threshold value ofthe high-resolution image read from the memory 17 and calculates theratio between the number of pixels in the high-brightness region and thetotal number (step S35), and the synthesis controlling part 45determines the position of a bending point (representing assigned symbol(A) in FIG. 14( a)) on a gradation converting curve for converting thetone level into an 8-bit tone level finally in accordance with therestrictions of the display device etc. (step S36).

At this time, for example, the distribution analyzing part 44 calculatesthe ratio of pixels having a tone level that is converted into apredetermined value (for example, a numerical value of 200) or more byconversion using a general gradation converting curve (represented bythe thick line in FIG. 14) for converting a 12-bit tone level into an8-bit tone level. When this ratio is larger than or equal to apredetermined threshold value, the synthesis controlling part 45 movesthe position of the bending point toward the side of smaller input tonelevels (left side in FIG. 14) in accordance with, for example, thedifference between the above-described ratio and the threshold value. Tothe contrary, when the above-described ratio is smaller than thepredetermined threshold value, the synthesis controlling part 45 movesthe position of the bending point toward the side of larger input tonelevels (right side in FIG. 14) in accordance with the difference betweenthe above-described ratio and the threshold value.

For example, when a subject is shot in a picture composition in whichbrightly glittering cloud occupies a significant proportion in theimage, the ratio of the high-brightness region obtained by thedistribution analyzing part 44 in step S35 becomes large and in responseto this, the position of the bending point on the gradation convertingcurve is moved nearer to the side of smaller input tone levels than theposition represented by symbol (A) in FIG. 14.

Next, the distribution analyzing part 44 calculates an average value oftone levels of pixels distributing in the range in which the tone levelis not saturated in the individual through images chosen as a target ofsynthesis in the above-described step S4 and exceeding the distributionrange of the tone levels in the high-resolution image (step S37), andthe synthesis processing part 35 determines the position of the upperlimit of the range of reproduced gradation (representing assigned symbol(B) in FIG. 14( a)) by the above-described gradation converting curvebased on the average value (step S38).

For example, for the through image obtained with one level underexposedand the through image obtained with two levels underexposed having thehistograms represented in FIG. 13, the average value of tone levels ofthe pixels distributed in the ranges (refer to FIGS. 13( c), (e))extracted respectively in the above-described step S3 is found, andfurther, in accordance with the average value, the upper limit of therange of reproduced gradation is moved nearer to the side of largerinput tone levels than the position represented by symbol (B) in FIG. 14(that is, upper limit of the 12-bit tone level). Here, when a pluralityof through images with different exposure values is obtained, it is onlyrequired to determine the upper limit of the range of reproducedgradation in accordance with a value obtained by further averaging theaverage values found respectively for each of them. At this time, it isalso possible for the synthesis controlling part 45 to move the upperlimit of the range of reproduced gradation to the position at which, forexample, the difference between the above-described average value andthe upper limit of the 12-bit tone level, that is, the upper limit ofthe tone level of the high-resolution image, is added to the averagevalue.

In this manner, a gradation converting curve is generated, with whichthe gradation in the high-brightness region can be reproduced by the8-bit tone level by compressing the gradation in the region having abrightness less than or equal to an medium level and expanding the rangein which the change of tone level in a high-brightness range isreproduced as the change of 8-bit tone level. The gradation convertingcurve is applied to the gradation converting processing of thesynthesized image obtained by synthesizing the through image and thehigh-resolution image obtained in the synthesis processing part 35 (stepS39).

In step S33, when a range is not extracted, in which a predeterminednumber or more of pixels are distributed in the above-described range inall of the through images, it may also be possible to determine thatthere is no feature of the gradation in the high-brightness region to bereflected in the high-resolution image and end the step withoutperforming synthesizing processing.

For the low-brightness region in the state of “saturated black” in thehigh-resolution image, it is also possible to make an attempt atexpansion toward the direction of lower brightness in the range ofreproduced gradation by applying a gradation converting curve having aplurality of bending points as represented in FIG. 14( b).

Next, synthesizing processing by the gradation synthesizing part 34 b isexplained.

The position alignment processing part 42 performs position alignmentbetween the high-resolution image and each through image obtained andbased on the result of aligning processing, the gradation synthesizingpart 34 b rearranges the brightness component and the color-differencecomponent of each pixel included in the through image in a pixel spacehaving the same density as that of the high-resolution image asrepresented in FIG. 8.

At this time, it is possible for the gradation synthesizing part 34 b toobtain a brightness component B′_(y) (i, j) of a rearranged image byrearranging the brightness component obtained by the above-describedgain correcting part 31 performing gain correction for each throughimage using the exposure value of the high-resolution image as areference value.

Next, when the high-resolution image has already been subjected to gammatransformation, the gradation synthesizing part 34 b performs inversegamma transformation on pixel data A_(y) (i, j) of the high-resolutionimage to find pixel data in a linear space and then finds a brightnesscomponent g′_(y) (i, j) of a synthesized image by performing weightedaddition the brightness component B′_(y) (i, j) of the rearranged imagealso in the linear space to a brightness component A′_(y) (i, j) of thehigh-resolution image having been subjected to the lineartransformation.

It is possible for the gradation synthesizing part 34 b to performweighted addition processing of a filter size m with the target pixel asits center represented in expression (2) by using, for example, aweighting function G (i, j, p). G (i, j, p) gives a heavy weight to thebrightness component B′y (i, j) of the rearranged image having a valueclose to tone level p of the target pixel (i, j) of the high-resolutionimage. By this weighted addition processing, the brightness componentg′_(y) (i, j) in the linear space of the synthesized image can be found.

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 2} \right\rbrack & \; \\{\mspace{79mu} {{G^{\prime}\left( {i,j,p} \right)} = {\exp\left( {- \frac{\left( {p - {B_{y}\left( {i,j} \right)}} \right)^{2}}{2\sigma^{2}}} \right)}}} & (1) \\{{g_{y}^{\prime}\left( {i,j} \right)} = \frac{\left\lbrack {{\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}\begin{Bmatrix}{G^{\prime}\left( {k,l,{A_{y}\left( {i,j} \right)}} \right)} \\{{{B_{y}^{\prime}\left( {i - \left( {m - 1} \right)} \right)}/2} +} \\\left. {{k - 1},{j - {\left( {m - 1} \right)/2} + l - 1}} \right)\end{Bmatrix}}} + {A_{y}^{\prime}\left( {i,j} \right)}} \right\rbrack}{{\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}{G^{\prime}\left( {k,l,{A_{y}\left( {i,j} \right)}} \right)}}} + 1}} & (2)\end{matrix}$

On the other hand, the gradation synthesizing part 34 b finds acolor-difference component A′_(CbCr) (i, j) of the high-resolution imageand a color-difference component B′_(CbCr) (i, j) of the rearrangedimage in a manner similar to the above based on the pixel level of thehigh-resolution image and the through image to be synthesized and findsa color-difference component g′_(CbCr) (i, j) of the synthesized imageby performing weighted addition of them.

At this time, in the high-brightness region in the “saturated white”state in the high-resolution image, the gradation synthesizing part 34b, taking into consideration that the color-difference component hassmall value, obtains component g′_(CbCr) (i, j) by weighting thecolor-difference component B′_(CbCr) (i, j) of the rearranged imagebased on the brightness component and adding it as represented inexpression (4). Add obtained component g′_(Cbcr) (i, j) and thecolor-difference component A′_(CbCr) (i, j) of the high-resolution imagewith a weight expressed by expression (3) (refer to expression (5)).

$\begin{matrix}\left\lbrack {{Expression}\mspace{14mu} 3} \right\rbrack & \; \\{\mspace{79mu} {{G^{\prime}(n)} = {\exp\left( {- \frac{n^{2}}{2\sigma^{2}}} \right)}}} & (3) \\{\mspace{79mu} {{g_{CbCr}^{\prime}\left( {i,j} \right)} = \frac{\left\lbrack {\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}\begin{Bmatrix}{G^{\prime}\left( {k,l,{A_{y}\left( {i,j} \right)}} \right)} \\{{{B_{CbCr}^{\prime}\left( {i - \left( {m - 1} \right)} \right)}/2} +} \\\left. {{k - 1},{j - {\left( {m - 1} \right)/2} + l - 1}} \right)\end{Bmatrix}}} \right\rbrack}{\sum\limits_{k = 1}^{m}{\sum\limits_{l = 1}^{m}{G^{\prime}\left( {k,l,{A_{y}\left( {i,j} \right)}} \right)}}}}} & (4) \\{{g_{CbCr}^{\prime}\left( {i,j} \right)} = {{{g_{CbCr}^{\prime}\left( {i,j} \right)}*{G\left( {255 - {A_{y}\left( {i,j} \right)}} \right)}} + {{A_{CbCr}\left( {i,j} \right)}*\left( {1 - {G\left( {255 - {A_{y}\left( {i,j} \right)}} \right)}} \right)}}} & (5)\end{matrix}$

With the arrangement, in the high-brightness region, it is possible torestore a natural color-difference component by reflecting thecolor-difference component B′_(CbCr) (i, j) of the rearranged image inthe color-difference component g′_(CbCr) (i, j) of the synthesized imagewith a heavier weight.

Similarly, also in the low-brightness region in the “saturated black”state, it is possible to restore a natural color-difference component byreflecting the color-difference component B′_(CbCr) (i, j) of therearranged image in the color-difference component g′_(CbCr) (i,j) ofthe synthesized image with a heavier weight.

Here, as represented in FIG. 8, the brightness component and thecolor-difference component of the through image to be synthesized withthe high-resolution image are mapped onto the rearranged image,respectively, and therefore, if the number of through images to besubjected to synthesizing processing increases, the number of pixelsmapped in the filter to be subjected to weighted addition when therearranged image and the high-resolution image are synthesized, and thusthe amount of information reflected in the pixel data of the synthesizedimage increases.

Consequently, by adjusting the filter size m to be applied insynthesizing processing in accordance with the number of through imagesto be synthesized, it is possible to reduce the burden of processingimposed on image synthesizing processing in the image processing devicewhile maintaining the quality of the synthesized image. For example,when the number of through images to be subjected to synthesizingprocessing is less than or equal to a predetermined number, a filtersize (for example, m=5), which will serve as a reference, is applied,and when the number of through images exceeds the above-describedpredetermined number, it is possible to suppress the influence of theposition alignment error on the quality of the synthesized image whilesuppressing the increase in the burden of processing due to the increasein the number of through images by reducing the filter size by onelevel.

By applying the gradation converting curve obtained by the distributionanalyzing part 44 and the synthesis controlling part 45 represented inFIG. 11 to the brightness component g′_(y) (i, j) of the synthesizedimage obtained as described above, it is possible to restore the changeof tone level in the high-brightness region in the “saturated white”state in the high-resolution image by utilizing information contained inthe through image and reproduce as a change of tone level (8-bit) aftergamma transformation.

As described above, according to the image processing device and theimage processing method of the present application, it is possible toselectively apply a proper synthesizing technique while taking intoconsideration various features including the distribution of tone levelin the high-resolution image and the low-resolution image to besubjected to synthesizing processing and the number of images to besubjected to synthesis. Consequently, with the technique to obtain ahigh-resolution synthesized image having an expanded dynamic range bysynthesizing the high-resolution image obtained by the present shootingand the through image, it is possible to properly reproduce thegradation, neither too much nor too little. For example, in a subjectsuch as cherry blossoms shot with a background of brightly glitteringcloud and blue sky can also improve image quality.

Embodiment 3

FIG. 15 represents a third embodiment of an image processing deviceaccording to the present application.

Among components represented in FIG. 15, that equivalent to each partrepresented in FIG. 11 is representing the symbol assigned representedin FIG. 11 and its explanation is omitted.

In a personal computer represented in FIG. 15, a CPU 48 and the memory17, a card reader 46, and a display processing part 47 are coupled via abus and the image processing device 25 is realized by a program thatcauses the CPU 48 to execute processing of each part of the imageprocessing device 25 represented in FIG. 11.

For example, it is possible to selectively apply a proper synthesizingtechnique and obtain a high-resolution synthesized image having anexpanded dynamic range while taking into consideration various featuresincluding the distribution of tone level in the high-resolution imageand the low-resolution image to be subjected to synthesizing processingand the number of images to be subjected to synthesis by recordingthrough images obtained with one level underexposed and two levelsunderexposed in the storage medium 30 along with the high-resolutionimage obtained by the present shooting by a digital camera, reading theimage data with the card reader 46 and storing it in the memory 17, andsubjecting it to the processing of the image processing device 25.

As described above, by allotting the image processing to synthesize thehigh-resolution image and the low-resolution image to the personalcomputer, the processing to be performed on the side of the digitalcamera can be limited to extraction and recording of image informationto be subjected to synthesizing processing, and therefore, it ispossible to enable a further faster operation of the digital camera. Inaddition, it is also possible for a user to confirm the result of imageprocessing described above based on the image displayed on a large,easy-to-see screen by subjecting the synthesized image thus obtained tothe display processing by the display processing part 47 and a displaypart 49 and providing it to the user.

As described above, it is possible to make an attempt to improve thequality of a synthesized image to be obtained finally by selectivelyapplying a proper synthesizing technique that has taken intoconsideration the features of the high-resolution image and thelow-resolution image to be subjected to synthesizing processing in thetechnique to cause the expansion of the range of reproduced gradationand the high speed of processing to coexist by synthesizing a pluralityof low-resolution images with different exposure conditions with thehigh-resolution image.

Embodiment 4

FIG. 16 represents an embodiment of an image processing device accordingto the present application.

Among components represented in FIG. 16, those equivalent to thecomponents represented in FIG. 1, FIG. 2, or FIG. 11 are showing thesame symbols assigned and their explanation is omitted.

In a digital camera represented in FIG. 16, light formed into a opticalimage on the imaging device 11 by the shooting optical system 12 whenshooting an image is converted into an electric signal in accordancewith its intensity by the imaging device 11 and is further convertedinto digital data by the analog/digital (A/D) converter 23 and stored inthe memory 17. The memory 17 represented in FIG. 1 is coupled to theimage processing device 25, the display part 29, the recording part 22,and the shooting controlling part 28 relating to the present applicationvia a bus and the shooting controlling part 28 switches the imagingdevice 11 to the reading mode.

The shooting controlling part 28 represented in FIG. 1 instructs thehigh resolution mode to the above-described imaging device 11 to readdata corresponding to all of the pixels in response to the operation ofthe release button by a user and the high-resolution image data obtainedfrom the electric signal read by the imaging device 11 in response tothis is stored in the memory 17 and, at the same time, subjected to theprocessing of the image processing device 25. On the other hand, beforeand after the release button is operated, the shooting controlling part28 switches the reading mode from the imaging device 11 to the throughimage mode and in response to this, the low-resolution image dataobtained by pixel skipping and pixel data accumulation within theimaging device 11 are subjected to display processing by the displaypart 29 via the memory 17 and the user is provided with informationabout the shooting range.

During the period of time in which the above-described through imagemode is applied, the low-resolution image data obtained from the outputsignal of the imaging device 11 obtained under various exposureconditions is stored in the memory 17 and subjected to the processing ofthe image processing device 25 along with the above-describedhigh-resolution image data. The through images stored in the memory 17in this manner are the low-resolution images obtained by shooting thesame subject as that of the high-resolution image obtained by thepresent shooting at reduced time intervals.

Hereinafter, the detailed configuration and operation of the imageprocessing device 25 are explained by taking a case as an example, whereauxiliary information is created for image synthesizing processing torestore to the original gradation in the saturated white region and thesaturated black region that appear in the high-resolution image obtainedby the present shooting using a low-resolution image obtained with anexposure value different from the correct exposure applied to thepresent shooting.

In the image processing device 25 represented in FIG. 16, a readprocessing part 50 reads the high-resolution image obtained by thepresent shooting executed in accordance with the release operation by auser and the through image stored corresponding to the high-resolutionimage from the above-described memory 17 and subjects them to theprocessing of the gain correcting part 31.

When the high-resolution image or the through image read by theabove-described read processing part 50 has already been subjected togamma transformation, the gain correcting part 31 first performs inversegamma transformation and then performs gain correction in accordancewith the difference in exposure condition between the high-resolutionimage and the individual through images. Specifically, the gaincorrecting part 31 performs gain correction by multiplying the tonelevel of each pixel included in the read through image by a constant inaccordance with the ratio between the exposure value appliedrespectively and the correct exposure, and a histogram creating part 52creates a histogram relating to the tone level for each image of theresult of gain correction.

As described above, it is possible to create a histogram L capable ofcomparison in a linear space with the distribution (refer to FIG. 13(a)) of pixels in a histogram H of the high-resolution image shot withthe correct exposure, instead of the histogram (refer to FIG. 13( b)) ofthe through image obtained with one level underexposed and the histogram(refer to FIG. 13( d)) of the through image obtained with two levelsunderexposed by creating the histogram of the through image after thegain correction (refer to FIGS. 13( c), (e)).

The histogram H of the high-resolution image and the histogram L of thethrough image obtained as described above are subjected to theprocessing of a saturated region detecting part 53 and the distributionanalyzing part 44 represented in FIG. 16, respectively.

The saturated region detecting part 53 analyzes the histogram H of thehigh-resolution image and finds, for example, the number of pixels thatexceed a threshold value corresponding to the upper limit of the tonelevel and the number of pixels that stay under a threshold valuecorresponding to the lower limit, respectively, and detects theoccurrence of the so-called “saturated white” or “saturated black”region in the high-resolution image based on whether or not therespective numbers of pixels exceed the respective predeterminedthreshold values, and notifies a selection processing part 55 of thedetection result.

The distribution analyzing part 44 represented in FIG. 16 analyzes thehistogram of the through image obtained as described above. Thedistribution analyzing part 44 searches for, for example, a range wheremore than a predetermined threshold number of pixels are distributed ina high-brightness range, which is equivalent to saturated level whencorrect exposure is applied (The high-brightness range is more than orequal to the tone level indicated by the broken line in FIG. 13). Thesearched result is notified to the selection processing part 55 asinformation that can be utilized for the restoration of the gradation inthe high-brightness region. The distribution analyzing part 44 searchesfor, for example, a range, such as those represented surroundings by thebroken line in FIGS. 13( c), (e), and when detecting such a range,notifies the detection result to the effect that information that can beutilized for the restoration of the gradation in the saturated whiteregion in the high-resolution image is included.

The selection processing part 55 represented in FIG. 16 chooses, when,for example, the saturated region detecting part 53 notifies that aregion in which the tone level is saturated is detected in thehigh-resolution image and the distribution analyzing part 44 notifiesthe detection result to the effect that the through image includesinformation that can be utilized for the restoration of the gradation inthe saturated white region in the high-resolution image, the throughimage that includes the above-described information as auxiliaryinformation used in image processing (synthesizing processing) for thehigh-resolution image, the main image.

The through image thus chosen and the high-resolution image, the mainimage, are compressed by an image compressing part 56, respectively, andsubjected to the processing of the recording part 22.

In the recording part 22 represented in FIG. 16, a header creating part57 receives auxiliary information including the through image compressedby the above-described image compressing part 56, creates headerinformation including the auxiliary information and shooting informationobtained from the shooting controlling part 28 via a bus, and subjectsit to the processing of an image file forming part 58.

As represented in FIG. 17, the image file forming part 58 stores thecompressed data of the main image received from the image compressingpart 56 in the image data part and to the compressed data, the headercreated as described above is affixed to form an image file and theimage file is written to and stored in a recording medium 26 via a writeprocessing part 59.

As described above, in the digital camera mounting the image processingdevice according to the present embodiment, when recording the mainimage in the recording medium, the above-described processing isperformed automatically and the through image having information usefulfor the restoration of the gradation in the high-brightness region orlow-brightness region of the main image is affixed as auxiliaryinformation to be subjected selectively to image processing.Consequently, a user is freed from the complicated selection andinstruction as to whether or not to perform restoring processing of thegradation by utilizing image synthesis for the region in which the tonelevel is saturated or which image to be subjected to synthesizingprocessing, and therefore, it is possible to reduce the work load of auser and the user can devote himself/herself to shooting.

In addition, as described above, it is possible to suppress the increasein the amount of data of the header part in the image file that storesthe main image by reducing the amount of data of the through image to besubjected to synthesizing processing by compressing it with the imagecompressing part 56.

Further, as described above, it is also possible to extract only thepixel data to be utilized directly in the synthesizing processing forthe purpose of the restoration of the gradation in the high-brightnessregion (or the low-brightness region), instead of compressing the entirethrough image to use it as auxiliary information, and to attach theextracted pixel data to the header of the image file as auxiliaryinformation.

For example, it is possible to extract features from the high-resolutionimage and the plurality of through images read from the memory 17 in themanner described in detail in the above-mentioned Embodiment 1 and thenextract the image data in the region of the through image correspondingto the region in which saturated white or saturated black occurs in thehigh-resolution image as auxiliary information after correcting thepositional shift between them based on the extracted features.

As represented in FIG. 18( a) with a hatch, when saturated white occursat the part of cloud and the sun captured in the high-resolution image,it is possible to considerably reduce the amount of data of auxiliaryinformation by extracting image data of the corresponding regions(represented in FIGS. 18( b), (c) with a hatch) of the through imageobtained with one level underexposed and the through image obtained withtwo levels underexposed and creating auxiliary information from theextracted image data and information about the above-describedpositional shift.

Further, when synthesizing the low-resolution through image and thehigh-resolution main image, it is also possible to carefully choose inunits of pixels the image data to be extracted from the through image asauxiliary information by taking into consideration that the rearrangedimage is formed by rearranging the through image in a pixel space withthe same density as that of the main image and that averaging processingusing a Gaussian filter with a predetermined size (for example, 5×5 or3×3) is performed for calculating the brightness value and thecolor-difference data of each pixel included in the above-describedsaturated white or saturated black region as represented in FIG. 8.

It is also possible to use, as auxiliary information, a high-resolutionimage obtained by shooting the same subject at a time slightly differentfrom that of the present shooting by applying a different exposurevalue, instead of a low-resolution through image. The amount of data ofthe high-resolution image is equivalent to that of data of the mainimage obtained by the present shooting, however, as described above, itis possible to suppress the amount of information to be affixed as aheader within a practical range by using only the image data in theregion to be utilized directly in synthesizing processing as auxiliaryinformation.

Further, it is possible for the header creating part 57 represented inFIG. 16 to insert various kinds of information obtained in the shootingstage into the header to be affixed to the data part of the image fileas auxiliary information, not limited to the image or part of the imageto be subjected to the above-described image synthesizing processing.

As described above, in secondary disclosed image processing device inthe section of MEANS FOR SOLVING PROBLEM, it is possible to form animage file including both the data of the main image to be subjected toimage processing and the auxiliary information by automatically choosingdata of other images that can be utilized effectively in imageprocessing by a personal computer and attaching it to the data of themain image as auxiliary information.

By recording such an image file in a recording medium, such as a compactflash memory card and an SD card, and subjecting it to sophisticatedimage processing by a personal computer etc., it is possible to maintainthe degree of freedom relating to image processing for image dataobtained by a digital camera etc. Further, the need of the selection andinstruction of a user as to whether or not to attach auxiliaryinformation at the time of shooting and which image data to be affixedas auxiliary information is obviated, and thus the burden of the usercan be reduced.

Embodiment 5

FIG. 19 represents an embodiment of an image processing method accordingto the present application.

Among components represented in FIG. 19, those equivalent to thecomponents represented in FIGS. 1, 2, and 15 are representing the samesymbols assigned and their explanation is omitted.

In a personal computer represented in FIG. 19, the CPU 48 is coupledwith the card reader 46 and the display processing part 47 via a bus andan image processing device 43 is realized by a program that causes theCPU 43 to execute processing of a read processing part 62, an auxiliaryinformation analyzing part 63, an image synthesis processing part 64,and an image correction processing part 65, to be described later.

The read processing part 62 represented in FIG. 19 reads an image file(refer to FIG. 17) stored in the recording medium 30 via the card reader46 and subjects the header part included in the image file to analyzingprocessing of the auxiliary information analyzing part 63.

The auxiliary information analyzing part 63 analyzes auxiliaryinformation included in header information, determines the kind of imageprocessing to which the auxiliary information is applied, and instructsthe read processing part 62 on the destination to which to send imagedata stored in the data part of the image file, and at the same time,delivering proper auxiliary information to the destination of the imagedata.

For example, when a through image with a different exposure value isrecorded as auxiliary information of the header part represented in FIG.17, the auxiliary information analyzing part 63 determines that it isthe auxiliary information to be used in synthesizing processing forrestoring the gradation in the region where the tone level of pixel issaturated and instructs the read processing part 62 to send image datato the image synthesis processing part 64, and at the same time, sendingthe through image data included in the auxiliary information to theimage synthesis processing part 64.

In response to this, the image synthesis processing part 64 performsimage synthesizing processing using the technique disclosed in theabove-described Embodiment 1, and thereby, it is possible to obtain ahigh-resolution synthesized image having an expanded dynamic range bycausing the image synthesis processing part 64 to automatically executeimage synthesizing processing by utilizing a proper low-resolution imagein accordance with the specification of the main image to be processed.

The synthesized image obtained in this manner is subjected to displayprocessing by the display processing part 47 and the display part 49 andprovided to a user, and therefore, it is possible for the user toconfirm the above-described result of image processing based on theimage displayed on a large, easy-to-see screen.

As described above, according to the embodiment of the image processingmethod of the present application, proper image processing isautomatically applied to the main image stored in the image file byutilizing auxiliary information based on the auxiliary informationincluded in the image file instructed to be read. Consequently, a useris required only to instruct to read the image file that stores the mainimage to be subjected to image processing and it is not necessary forthe user to perform complicated task, such as to specify the image filethat stores another image to be synthesized with the main image.

Similarly, it is possible to subject the image data and auxiliaryinformation included in the image file to proper image processing bydiscriminating the auxiliary information applied to various kinds ofimage processing with the auxiliary information analyzing part 63.

As represented in FIG. 19, by causing a computer to execute the programthat realizes the secondary disclosed image processing method in thesection of MEANS FOR SOLVING PROBLEM, it is possible to perform properimage processing for the main image data by utilizing the auxiliaryinformation read from the image file recorded by the image processingdevice having the configuration represented in FIG. 16.

At this time, it is possible for the user to perform image processingusing proper auxiliary information only by specifying the image filethat stores the image data to be subjected to image processing, andtherefore, the workload of the user can be reduced.

The present application can be embodied in other forms without departingfrom the spirit and its essential features. Therefore, theabove-described embodiments are only examples in all respects and shouldnot be interpreted as limitative. The scope of the present applicationis indicated by the scope of claims and not restricted by the presentspecification. Further, all of the modifications and alterationsincluded in the equivalent scope of claims should be included in thescope of the present application.

INDUSTRIAL APPLICABILITY

As described above, according to the image processing device, the imageprocessing method, and the image processing program of the presentapplication, it is possible to obtain an excellent effect for expandinggradation by synthesizing a high-resolution image and at least onelow-resolution image, and therefore, the present application is veryuseful in an image processing device incorporated in an electroniccamera and an image processing device realized by causing a personalcomputer to execute an image processing program.

In particular, when a high-resolution image and a low-resolution imageobtained by changing exposure conditions are used by applying it to anelectronic camera, it is possible to shorten the period of time duringwhich a user has to fix a camera angle and enable shooting withexpanding gradation more easily than before by utilizing that the readtime of the low-resolution image is short.

Further, when applying a technique to synthesize a high-resolution imageand a low-resolution image, it is possible to improve the quality of ahigh-resolution image having an expanded dynamic range obtained bysynthesizing processing by properly controlling a technique tosynthesize a low-resolution image. Consequently, it is made possible toapply a technique to expand gradation by image synthesis in applicationsin which a high image quality is required.

Furthermore, by providing a technique that considerably reduces workloadof a user when applying image processing with high degree of freedomusing a personal computer to image data obtained by an imaging device,such as a digital camera, it is made possible to provide an image ofhigh quality to which a sophisticated image processing function has beenapplied even to a user who tends to avoid complicated operations andremarkably improve the service level for the user, and therefore, thisis very useful in the field of an imaging device and image processingdevice, such as a digital camera.

1. An image processing device, comprising: an image inputting parttaking in at least one of a plurality of low-resolution images and ahigh-resolution image obtained by shooting the same subject whilechanging exposure condition; a shift detecting part detecting apositional shift of pictorial pattern between each of the plurality oflow-resolution images and the high-resolution image; and a gradationexpanding part generating a synthesized image in which a range ofreproduced gradation is expanded by performing position alignmentbetween each of the plurality of low-resolution images and thehigh-resolution image based on the positional shift, extractinggradation information of the plurality of low-resolution images, andsynthesizing the plurality of low-resolution images with thehigh-resolution image.
 2. The image processing device according to claim1, wherein the image inputting part takes in two or more low-resolutionimages; and the gradation expanding part synthesizes the gradationinformation of low-resolution represented by each of the plurality oflow-resolution images with the high-resolution image in a multiplexingmanner.
 3. The image processing device according to claim 1, wherein theshift detecting part comprises: a phase dividing part generating aplurality of pieces of sampling information with sample positionsshifted from one another by extracting an edge component of pictorialpattern from the high-resolution image and performing sub-sampling ofthe edge component while shifting phases; and a precisely detecting partdetecting a positional shift with a finer precision than a pixelinterval of the low-resolution images by detecting the positional shiftwith which the pictorial pattern best matches between each of theplurality of low-resolution images and the plurality of pieces ofsampling information.
 4. The image processing device according to claim1, wherein the gradation expanding part: determines a high-brightnessregion and a low-brightness region of the high-resolution image;increases a synthesis ratio of one or more low-resolution imagesunderexposed as to the high-brightness region of the high-resolutionimage; and increases a synthesis ratio of one or more low-resolutionimages overexposed as to the low-brightness region of thehigh-resolution image.
 5. The image processing device according to claim1, wherein the gradation expanding part comprises: an adjusting unitadjusting a tone level of a corresponding pixel of a synthesized imageby reflecting a tone level of each pixel included in at least one of thelow-resolution images chosen from the plurality of low-resolution imagesand a tone level of corresponding pixel of the high-resolution image;and a controlling unit controlling adjusting processing of the tonelevel of each pixel of the synthesized image by the adjusting unit basedon at least one of the followings, a histogram of the tone levelobtained for each of the plurality of low-resolution images, a histogramof the tone level obtained for the high-resolution image, and the numberof the low-resolution images acquired by the image inputting part. 6.The image processing device according to claim 5, wherein thecontrolling unit comprises: an analyzing unit analyzing a distributionof pixels in a predetermined range of tone levels as to the histogram ofthe tone level obtained for each of the plurality of low-resolutionimages and the histogram of the tone level of the high-resolution image;and a converting curve fitting unit adjusting a gradation convertingcurve used to confine the tone level of each pixel of the synthesizedimage within a predetermined range in the adjusting unit based on theanalysis result by the analyzing unit.
 7. The image processing deviceaccording to claim 5, wherein the controlling unit comprises: ananalyzing unit analyzing a distribution of pixels in a predeterminedrange of tone levels as to the histogram of the tone level obtained foreach of the plurality of low-resolution images and the histogram of thetone level of the high-resolution image; and a choosing unit choosing atleast one of the low-resolution images to be used in adjustingprocessing by the adjusting unit based on the analysis result by theanalyzing unit.
 8. The image processing device according to claim 5,wherein the controlling unit comprises a range determining unitdetermining the size of a region of low-resolution images to bereflected in the adjustment of the tone level of each pixel included inthe synthesized image by the adjusting unit in accordance with thenumber of low-resolution images to be used in synthesizing processing bythe adjusting unit.
 9. The image processing device according to claim 5,wherein the controlling unit comprises: a brightness weight determiningunit determining a brightness weight to be applied when reflecting abrightness component, which corresponds to the pixel of at least one ofthe low-resolution images chosen, to a brightness component of eachpixel included in the synthesized image by the adjusting unit; and acolor-difference weight determining unit determining a color-differenceweight to be applied when reflecting a color-difference component, whichcorresponds to the pixel of at least one of the low-resolution imagechosen, to a color-difference component of each pixel included in thesynthesized image by the adjusting unit, wherein the color-differenceweight determining unit comprises a weight adjusting unit adjusting avalue of the color-difference weight in accordance with the magnitude ofa color-difference component corresponding to each pixel of thehigh-resolution image.
 10. An electronic camera, comprising: the imageprocessing device according to claim 1; and an imaging part shooting asubject with at least two kinds of resolution, wherein thehigh-resolution image processed by the image processing device is astill image of the high-resolution shot by the imaging part, and theimage processing device processes at least one of the plurality oflow-resolution images shot by the imaging part before and/or after theshooting of the still image with the exposure condition different fromthe high-resolution image.
 11. The electronic camera according to claim10, comprising a monitor part displaying an image, wherein the imagingpart: sequentially shoots low-resolution through images, through imagesare images obtained by pixel skipping to provide moving images for viewfinder, and displays a moving image on the monitor part; and shoots aplurality of low-resolution images with timing not synchronizing theshooting of the through images and under the exposure conditiondifferent from the high-resolution image.
 12. An image processingdevice, comprising: a determining unit determining whether or not toattach one or more pieces of other image data as auxiliary informationwhen performing image processing to an image data based on the imagedata capturing an image of a subject shot by an imaging unit and apredetermined condition; and a recording unit putting together a mainimage data, which is the image data to be processed in the imageprocessing, and the auxiliary information into one image file andrecords in a recording medium in accordance with the determinationresult of attaching the auxiliary information.
 13. The image processingdevice according to claim 12, wherein the determining unit comprises asaturation detecting unit notifying the recording unit of thedetermination result to attach the auxiliary information when detectinga region in which a tone level of pixel included in the main image datais saturated.
 14. The image processing device according to claim 12,wherein the recording unit comprises a first choosing unit choosingother image data, obtained by shooting the same subject as the mainimage data at minimal time intervals, as the auxiliary information andused in process of attaching to the main image data.
 15. The imageprocessing device according to claim 12, wherein the recording unitcomprises a second choosing unit choosing other image data, obtained byshooting the same subject as the main image data at minimal timeintervals under a different shooting condition, as the auxiliaryinformation and s used in process of attaching to the main image data.16. The image processing device according to claim 12, wherein therecording unit comprises a third choosing unit choosing image data inwhich a distribution range of data tone level of pixel represented in ahistogram has a predetermined relationship with a peak position in ahistogram of the main image data among other images, obtained byshooting the same subject as the main image data at minimal timeintervals under a different shooting condition, as the auxiliaryinformation and used in process of attaching to the main image data. 17.The image processing device according to claim 12, wherein the recordingunit comprises a header creating unit creating header informationincluding the auxiliary information and attaching the header informationto the main image data.
 18. The image processing device according toclaim 17, wherein the header creating unit comprises an extracting unitextracting part of other image data used in image processing of the mainimage data in accordance with the purpose of the image processing andproviding the part of other image data extracted in header informationcreating processing as auxiliary information.
 19. An image processingmethod, comprising: an image inputting step for acquiring a plurality oflow-resolution images obtained by shooting the same subject under aplurality of exposure conditions with an imaging unit shooting images ofthe subject with a plurality of different resolutions and ahigh-resolution image obtained by shooting the subject under a correctexposure condition with the imaging unit; and a synthesizing step forsynthesizing at least one of the low-resolution images chosen from theplurality of low-resolution images and the high-resolution image, andgenerating a synthesized image having a resolution equivalent to thehigh-resolution image, wherein the synthesizing step comprises: anadjusting step for adjusting a tone level of a corresponding pixel ofthe synthesized image by reflecting a tone level of each pixel includedin at least one of the low-resolution images chosen and a tone level ofcorresponding pixel of the high-resolution image; and a controlling stepfor controlling adjusting processing of the tone level of each pixel ofthe synthesized image in the adjusting step based on at least one of thefollowings, a histogram of a tone level obtained for each of theplurality of low-resolution images, a histogram of a tone level obtainedfor the high-resolution image, and the number of the low-resolutionimages acquired in the image inputting step.
 20. An image processingmethod for: reading main image data and auxiliary information used inimage processing from an image file including the main image data, whichto be processed in an image processing, recorded in a computer-readablerecording medium; and executing the image processing for the main imagedata using the read auxiliary information.
 21. A computer readablemedium storing an image processing program capable of instructing acomputer to function as the image processing device according to claim1.